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OVERVIEW 

‘Is utilizing a mobile application to track student competency achievement efficient and 
effective?' Thus began my presentation at the 201 1 CNIE Conference in Hamilton, Ontario 
which was themed ‘Cascades of Innovation’. 

First - a bit of background. Over 8 years ago, the company I now work for in Edmonton, 
Alberta, called Great Big Solutions Ltd., launched and now continually develops and supports 
a web and mobile competency tracking solution called CompTracker^. The solution has so far 
been tailored to post-secondary health care students tracking many different skill and 
competency profiles in their studies. 

Though many people still haven't heard of us, personal experience and market information 
tells us demand is going to continue for sound, mobile applications like ours that deliver on the 
promise of making our work and home lives easier. 

In her 2005 article. Enabling Mobile Learning, Ellen Wagner said “The use of PDA-based 
performance fools fo supporf classroom insfrucfion and on-fhe-job fraining alike has been well 
under way for a number of years, particularly in the fields of medicine and allied health, 
business, and journalism..." Further, “...whether we like it or not, whether we are ready for it or 
not, mobile learning represents the next step in a long tradition of technology-mediated 
learning.” 1 

So we know there's demand, and we see it increasing, but, is this a good thing? This question 
really stems from my personal need to know, and to question, almost daily, ‘Am I making a 
difference and contributing to a greater good through the work that Ido?’ 

Indulge me for a moment as I relay a story. 

Picture this: 

A 20-something student on the last day of her practicum, just before graduation. She's a 
province away from her home university, and only a few junior staff are still onsite before the 
weekend. 

Carefully taking 20 plus pages of her reflective journals, and detailed self and preceptor 
evaluations, she approaches the fax machine. Fanning the pages, she places them face 
down on the fax, then double and triple checks the number. Pressing ‘Send,’ she holds her 
breath for a moment, but relaxes as the pages begin to scan easily. She leaves the fax 
humming away, and goes to bid farewell to staff and her mentors of the past four months. 

1 5 minutes later, the student returns to the fax and her heart leaps into her throat! She looks at 
the crumpled pages and jammed fax machine in utter disbelief. How was it possible to make 
such teeny, tiny accordion folds in paper?! With some pages now illegible and torn - she 
wishes she'd made copies of her evaluations first - but too late!! 
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Utterly dejected, she cleans the fax as best as she can and places a desperate call to her 
practicum coordinator hundreds of kilometres away. (True story about the author!) 

So experience, then and now, leads me to believe that mobile competency tracking is a 
viable solution. But what do educators and our customers believe? 

A Survey is Born 

An article review on the ‘efficacy of mobile competency tracking’ revealed there was limited 
study on this specific topic. I was able to find a number of publications, however, that 
discussed various facets of mobile technology incorporated as a learning/teaching tool both 
in and outside the classroom. 

Given the lack of formal studies available through an online search, the time also seemed 
right to survey our customers about electronic competency tracking to get to the heart of 
what they found beneficial-or not. Bearing in mind that some of our students may use our 
website, a device and sometimes a blend of both to submit competency information, some 
questions remain. The purpose of this article is to present some food for thought and potential 
points for future study. 

The CompTracker ® Benefits Assessment Survey was designed to be brief and anonymous, and 
was provided to customers who had been utilizing our application for at least one year or 
more. It included a mix of 1 5 demographic, open-ended, multiple choice, rating scale, and 
modified semantic differential questions. 

Following the article review and survey, the following four areas surfaced as the most 
significant. That is, through utilizing mobile technology: 

1 . Students may be more focused, organized and better ‘behaved’ 

2. Feedback and collaboration may be improved 

3. It provides a good return on investment due to its Swiss Army Knife capabilities 2 

4. It results in decreased workload and increased job satisfaction for educators 

Students Wired for Technology 

If you as an educator or parent have sensed some differences in terms of how quickly your 
students or children have glommed onto mobile technology as a learning tool versus your own 
speed of adaptation, there may be good cause. As early as 2001 , Mark Prensky coined the 
term ‘Digital Natives' 3 to refer to the Kindergarten through College age group who not only 
behave differently in interacting with technology-their brains may physically be wired 
differently. 4 So if you sometimes cannot relate to your kids or students in this aspect it may be 
because they've embraced mobile technology-literally! body and soul. As ‘Digital 
Immigrants' 3 we may lag behind in adopting this technology to the same comfort level or 
extent, but it appears here to stay and there are good reasons for us to jump on the 
bandwagon with this more technically adept demographic. 
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1. Students more focused, organized and better ‘behaved’ 

A Norris and Soloway article talks about using handheld computers to bring K-12 classrooms 
into the 21st century, but central to success, they say, is that students be sole users of a given 
device. 5 As sole users, students don't have to share a bank of laptops or mobile devices, let 
alone, computer lab time, and teachers then seem more likely to incorporate learning 
activities in the classroom that are centered around the use of technology. In turn, 
incorporating technology into learning activities may then aid student focus and organization. 
From this same Norris and Soloway article: 

• A fourth-grade student said “I don't have to hunt for pieces of paper anymore... 
Everything is in one place.” 

• Kerri Neubauer, instructional technology coordinator, Alvin School District in Texas 
said “...And we see that the use of the devices definitely results in increased student 
motivation. We are seeing significantly fewer behavior problems in classes where the 
mobile computers are being used.” 

• Monique Shorr, sixth-grade science and social studies teacher at Hartland Farms 
Intermediate Schools in Michigan: “...Over the semester my students become 
autonomous learners who can take charge of their own learning.” 

From Ellen Wagner's article mentioned earlier, “Technology in and of itself may not guarantee 
better learning , but when effectively deployed, technology can help focus attention while 
attracting and maintaining a learner’s interest .” 1 

In post-secondary circles, educators may not run into as many behavior or distractibility issues 
as K-l 2, but some of our CompTracker® customer survey comments were similar in theme to 
those above: 

• Students are able to track their progress and [have] been more time management 
efficient. 

• Greater clarity by students as to program expectations. 

• Students are happy not to have to drag a paper binder with them everywhere they 
go. 

• No outstanding question from faculty as to which students complete the 
competencies. 

• THE simplest method of competency tracking we have utilized in the last 1 1 years. 

In the Internet Journal of Internal Medicine, 201 0; Ferenchick, Sneed, Solomon and Mohmand 
evaluated “The effect of a grading incentive and a problem-specific mobile electronic 
clinical evaluation tool (eCEX) in the direct observation of medical student’s clinical 
competencies: A pilot study.” 6 
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In the study, authors say, “Direct observation of medical students’ clinical skills by faculty is 
uncommon.’’ This is also the case for many of our customers' health care students who are 
often evaluated by field or clinical site preceptors. Regardless of who is evaluating the 
student, the study suggests that transparency of information is important, and goes on to 
conclude, “Students and the faculty both agreed that the CEX helped them understand 
which problem-specific competencies were targeted for fhe assessmenf... Faculty generally 
agreed that eCEX improved their ability to provide feedback to the students.’’ Which ties in 
neatly to the next point... 

2. Improved collaboration and feedback 

I think we can agree that timely feedback serves a very important purpose for students. It 
provides an immediate benchmark of their performance. They can also refer to feedback 
given to them over time to gauge their progress and relating back to point number one in this 
article, it can help them determine where to focus their efforts and improve. 

From the Educause Learning Initiative site, under M-Learning and Mobility: 

Using portable computing devices (such as laptops, tablet PCs, PDAs, and smart 
phones) with wireless networks enables mobility and mobile learning, allowing teaching 
and learning to extend to spaces beyond the traditional classroom. Within the 
classroom, mobile learning gives instructors and learners increased flexibility and new 
opportunities for interaction. Mobile technologies support learning experiences that are 
collaborative, accessible, and integrated with the world beyond the classroom . 7 

With CompTracker®, there are numerous opportunities to provide feedback and to 
collaborate through the application. Students are generally the ‘drivers' of the system, and 
their submissions may be almost real time: as soon as they submit information on the website or 
sync their device via wireless, the information may be reviewed, feedback given, and where 
appropriate, graded. The entire submission, including feedback and rating then becomes 
part of the students' educational profile and achievement record. 

In the case of a significant skills deficit, the practicum coordinator or preceptor may even 
initiate an Action or Learning plan through CompTracker® to address the skill shortfall. This 
shortfall will be evident to all parties with the appropriate permissions to view the students' 
information and may be verified and audited later. This near real-time process means that 
issues may be identified sooner rather than later and appropriate measures put in place 
before it may be too late. 

From our CompTracker® Benefits Assessment Survey, these were some comments received 
about feedback and collaboration: 

• ... more transparent for fhe sfudenf; fhey can access fhe information at any time; 
easier to follow fhe students from one course to another... 

• ... students can plan their clinical activities according to their reports. 
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• Use of CompTracker has allowed for more timely and accurate evaluation of our 
students in the Clinical and Practicum setting; we would not switch back to paper. 

The CompTracker® website also supports asynchronous messaging which can be utilized by 
faculty, students or preceptors to initiate record specific or general queries. These message 
notifications are sent to the users' usual email account, and with email capability on their cell 
phone, they will be notified of this new message which may also be flagged as urgent and 
requiring their immediate attention. 

Whether utilizing a device or the website, the transparent evaluation and feedback process 
helps to keep the lines of communication open, and parties up to date as well as 
accountable. And rather than detracting from collaboration, I am coming to accept that in 
this new digital age that face-to-face interaction isn't always necessary to support teamwork 
on a given endeavour such as when we’re preparing students for competency in entry-to- 
practice skills. 

3. Good ROI with ‘Swiss Army Knife’ Capabilities 2 

With textbooks, references, calendars, email, mobile web 
browsers, document storage, and communication all in one, a 
mobile device seems to deliver a good ‘bang for your buck' 
whether you're a student buying your own or a school district 
administrator making a bulk purchase. Now I'm not sure I would 
want to read a 500 page text on my iPod Touch if I was that 20 
something student again, but given the choice to read it on an 
iPad or other type of tablet and have the benefits of mobility-no 
question-1 would have saved the minimum wage earnings from 
my student summer job to purchase one. 

There are currently more than 425,000 applications for free or a 
fee from the Apple® App Store: 
http://www.apple.com/iphone/apps-for-iphone/ . 

Apple® also has a site specifically outlining education 
applications: http://www.apple.com/education/apps/ . 

Norris and Soloway refer to a New York Times article from May 2007 and the one-to-one laptop 
initiatives being terminated by some school districts due to the total cost of ownership. Back in 
2004, Beverly Walker, deputy superintendent, Alvin (Texas) Intermediate School District 
commented, “If Alvin was going to prepare its children for the 21st century, I had to find a way 
to provide each child with a computer.’’ She goes on to say, “I didn't see the laptop costs 
would come down enough to accomplish the goal. But handheld computers seemed to have 
the right price-performance ratio.” 5 
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In this same article, Norris and Soloway comment, “Cell phones or cell phone use is banned 
from many school districts now, but savvy administrators will realize they can avoid buying 
computers since the students’ own devices will be sufficient for most learning tasks." 5 

Though return on investment depends on many factors, strongest being the type of device 
and the extent to which the device is used in the educational setting, when compared to the 
cost of purchasing a laptop it’s hard to argue with the return. A laptop will also have limited 
mobility. With a truly mobile device such as an iPod Touch, students can be behind the 
bulkhead in the X-Ray department, or on wards observing rounds, and when the time is 
appropriate they can still track their competencies, obtain written feedback and get a valid 
electronic signature from their preceptor. As soon as they sync their device via wireless, their 
placement coordinator from any Internet-connected browser will know what they've been 
working on, how well they're doing, and if any intervention is required. Oh - and no jammed 
fax machines to contend with! 

Given the portability, capability, and accessibility of mobile competency applications, the 
bottom line on ROI is difficult to dispute. 

4. Workload and Job Satisfaction 

My informal research indicates workload and job satisfaction are positively impacted through 
the use of mobile technology. With my customers and on a personal note, with many of my 
extended family working, or having worked as educators, I was very pleased with these 
findings. 

From our CompTracker ® Benefits Assessment Survey: 

• We need current data on a student’s competency status and how they are 
progressing. Paper is too long and takes many man hours to tally. 

• .... Our gradebook used at the college requires overnight results. It is easy for me to 
get the results to input my grades on our college system. 

• Allows time forme to ‘teach.’ 

• Makes more efficient use of time in a fast-paced program, and also it is recognized 
by our accrediting body as an acceptable way of tracking and recording student 
progress. 

In our survey, we specifically questioned customers about the impact to their workload. We 
asked, “How has CompTracker Reporting affected the time you spent previously monitoring or 
calculating student progress when it was done on paper?” Refer to Figure 1 for the results. 
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Figure 1 

To summarize this chart, 
58% of survey 
respondents 

experienced a workload 
reduction of at least 30% 
(the orange, pale blue, 
pink and pale green pie 
pieces). An equal 
number of respondents 
(14%) saved either 10- 
20% or 20-30%. Only a 
small minority of 1 5% of 
respondents said that 

they saw “No noticeable reduction” or only up to 5%. 

0-5% is a very low rate of workload savings, and even 1 0-20% seems low. Based on comments 
made in other parts of the survey, there may be some reasons to account for this. 

First, some customers, due to their role, use only one component of our system, and they find 
no time savings when electronically marking attendance, for example. One customer 
commented it took them more time as compared to doing it on paper. This is understandable 
when nothing further needs to happen with that attendance record. If that same paper 
record needs to be filed or entered by someone electronically in another system, however, 
and they are not one in the same person gathering and entering the information, they may 
not experience the time savings. 

Some of our customers are also still quite dependent on paper tracking methods. They may 
even duplicate some of their efforts by validating CompTracker® data against paper records. 
As comfort and confidence utilizing an electronic competency tracking system increases, the 
amount of paper that needs to be utilized or validated should decrease and result in lower 
workloads. 

From the Blog, Reducing Teacher Workload by Mark Bethelemy, “Talk to most teachers in 
England... about what would make the biggest improvement to their professional lives and 
they will say ‘reduce the paperwork .”' 8 
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The question, “Has CompTracker affected your personal job satisfaction?” produced the 
following results; see Figure 2 below. 
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Figure 2 

To summarize, a 
convincing 59% of 
respondents rated their 
job satisfaction at least 
a 4 out of 5 with 5 being 
“Significant positive 
impact.” At the other 
end of the scale, 1 5% of 
customers reported they 
were “Not noticeably” 
affected; a rating of 1 . 



SUMMARY 

Mobile competency tracking is not without its challenges, but utilizing what this technology 
has to offer both in and outside the classroom may lead to improved student behavior, and 
can assist learners as well as educators to stay better organized and focused on requirements 
and outcomes. 

Mobile technology readily facilitates feedback. For ‘Digital Immigrants,' it presents us with an 
alternative method to face-to-face collaboration which we may want to consider embracing 
to the same extent as younger generations for the convenience and benefits it can offer. 

With its many capabilities, its portability and accessibility, mobile technology is cost effective. 
Depending, perhaps, on your role and also the extent to which you need to manage paper 
and then extrapolate information from it in the future, it can be quite time and workload 
effective as well. 

With 78% of our survey respondents indicating their job satisfaction rated at least 3 out of 5 as a 
result of using our solution, and taking the other factors in this discussion into account, the 
answer to the question, ‘Am I making a difference and is electronic competency tracking 
efficient and effective?' is a Great Big Solutions - YES! 
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